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In this paper we describe some of our experiences in conducting the pilot 
test of the CSE Elementary School Evaluation KIT: Program Planning and dis- 
cuss some of our tentative conclusions arising from these experiences. The 
conclusions are based on the premise that early formative evaluation of prod- 
ucts can provide the developer with user information to guide the further 
conceptualization of the product. Therefore » we believe that pilot tests 
should be scheduled soon after the developer has made his first provisional 
decisions about the content, sequence, and method of presentation. 

Whereas formal, preplanned tests to gather quantatative data may be more 
appropriate for later stages of product development, informal , cl inical pro- 
cedures and qualitative data should be considered at the early stage where 
the information will be used to guide development. 

Specific skills that we found to be important in conducting such early 
pilot tests can be termed communication skills* But they are not those com- 
munication skills in which product developers are usually trained--that is, 
skills for transmi tting informaticn. Rather, they are the skills involved in 
receiving information: observing and listening to people, ascertaining the 
meaning of what they are saying or doing, and teasing out the implications of 
that meaning for the future development of the product. Although there are 
difficulties in establishing communication between developer and practitioner, 
we believe the advantages of producing a product responsible to user needs 
make the attempt to analyze and deal with them worthwhile. 

In this paper, we would like to take you with us through the experiences 
which led us to this conclusion. We will begin by describing the Program 
Planning KIT in its current national field test form. We will then go back 
to one of our first pilot tests--when the materials were in quite different 
fonn--and describe to you what happened with one pilot test team. We will 
indicate to you what we learned about the KIT and what we learned 



about the process of communication. After restating our conclusions, we 
win outline the similarities between our position and the concerns of 
particular writers in related areas. 

The CSE Program Planning KIT 

The CSE Elementary School Evaluation KIT: Program Planning is the sec- 
ond in a series of four self-instructional packages designed to help ele- 
mentary school staffs proceed from needs assessment and program planning 
through formative and summative evaluations. The first KIT* provides pro- 
cedures for ranking goal statements, assessing student performance on highly 
ranked goals, and making decisions about high priority needs. The third and 
fourth KITs, currently under development, provide procedures to help school 
principals collect and analyze information to make decisions about program 
improvement or program continuation. 

The Program Planning KIT is intended to provide an elementary school 
principal and a planning team of teachers with the structure and resources 
to plan for programs and their evaluations. At the end of the program plan- 
ni^ng process, the school staff will have produced a written program plan con- 
taining a description of the instructional, evaluation and management com- 
ponents of a program which can be implemented in their school. The KIT can 
be viewed as a self-administored inservice training program conducted at the 
school. It is anticipated that the first time the KIT is used, teachers will 
be simultaneously learning skills and applying them in planning their own 
program. In subsequent planning cycles » it Is more likely that the KIT will 
be used as a resource rather than to provide instruction. 



*CSE Elementary School Evaluation KIT: Needs Assessment. (Boston: 



Allyn & Bacon, 1972). 
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The KIT currently consists of a number of components: 

The Coordinator's Handbook helps the principal (or his designate) to 
initiate the planning effort. It provides guidelines for facilitating 
the work of the Planning Team and the Evaluation Planner; for develop- 
ing a management system for both the program and the evaluation; for 
writing the program plan; and for institutionalizing on-going planning 
and evaluation in the school. 

The Planning Team materials consist of a Procedures Booklet providing 
step-by-step instructions to a teacher planning team for eanh part of 
the planning process, a Resource Book containing additional informa- 
tion and practice exercises, a Chronicle in which team records are 
kept, and Leader Guide Cards to assist a team leader in preparing for 
and conducting team meetings. Each planning team chooses from one of 
three planning strategies--Objectives-Based, Teaching Models, or 
Materials-Based— and uses the set of materials developed for that 
strategy to produce a written program plan. 

The ^valuator's Guidebook provides the individual acting as Evaluation 
Planner with a general orientation to evaluation. Evaluation Planner 
Guide Cards, keyed to the Procedures Booklet of each strategy, give him 
detailed suggestions for conducting discussions which lead the team to 
decisions about how to evaluate their program. 



Pilot Testing of the CSE Program Planning KIT 

At the Center for the Study of Evaluation, we regard evaluation as "the 
process of determining the kinds of decisions that have to be made; selecting, 
collecting and analyzing the information needed in making these decisions; 
and reporting this information to appropriate decision makers*' (Klein, Fensterma- 
cher, & Alkin, 1971, p. 9). In agreement with Baker and Alkin (1973), we 
accept the view that formative product evaluation consists of activities that 
will provide developers with the information they need to modify and improve 
their products. We think of pilot testing as a very early stage of formative 
evaluation during which first draft components of the product are tried out 
with a sample of users. 

Early in 1973, we started making plans for the pilot test of the Program 
Planning KIT. Although we had begun to conceptualize the KIT, only parts of 
it were written. An earlier formulation of the KIT took the form of a set of 
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procedures to help lo:al elementary school principals select ''the best avail- 
able" instructional program (materials) for achieving pre-specif ied goals. 
But by the time we began to plan for our pilot tests, our conception of the 
KIT had broadened cons* derably . We had decided that a planning team of teachers 
should be the major pa'^ticipants in instructional planning, with the principal 
in a facilitating role. Instead of a single planning strategy, there now were 
three alternative planning strategies. One was a modification of the orig- 
inal strategy for selecting a program, A second strategy used behavioral ob- 
jectives as the building blocks of the program; the third used conceptions 
of teaching and learning (teaching models) as the basis for constructing the 
program. 

At the time of the pilot test, we were committed to the idea that the 
purpose of the KIT was to help school level planners produce a written plan 
that would describe an educational program so as to facilitate both imple- 
mentation and eventu-il evaluation activities. We felt we had identified some 
of the tasks related to achieving this broad purpose. We did not, however, 
have a final commitment to the content, sequence or form of the KIT; we 
felt that potential users cou'id help us make these decisions. 

Our initial plans for the pilot test were similar tc the plans produced 
by other institutions who develop instructional products. We were particu- 
larly interested in schools that served minority students. We wanted plan- 
ning teams of teachers who would agree to use the strategies and materials 
we provided them; to spend the planning time suggested in the KIT; and to 
adhere to the suggested schedule. We wanted participants who would complete 
our tests, questionnaires, and submit to us a written copy of the program 
plan they would produce. We also planned to have at least one CSE staff mem- 
ber attend every planning session as an observer. In short, we thought our- 
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selves receptive to the ideas of participants in the pilot test, but we 
wanted to control the conditions of the pilot test and the collection of 
data. 

To identify the schools that would be included in the pilot test, we 
contacted school officials in a nearby district and were given suggestions 
as to principals likely to participate. In the period between January and 
April of 1973 we phoned fifteen principals, received invitations to speak 
to teachers at seven schools, and were finally able to get commitments from 
teachers in three schools (who formed a total of eight planning teams) to 
pa'f^ticipate in the pilot test. Because the time required of the teachers 
was considerable (15-24 hours), in order to recruit participants, we had to 
make arrangements for district in-service salary credits. 

When we scheduled our pilot tests, we did so with some misgivings. We 
cou^d have used more time to prepare our first draft materials but our pro- 
ject time line was forcing us to act. We also felt that conducting the 
pilot tests before we thought ourselves ready might have some benefits. 
First, it would create production deadlines. Second, it would force us to 
justify our ideas in concert with teacher teams. Third, we reasoned that we 
could remain more open to teachers* ideas about revision and subsequent de- 
velopment if we were wot committed to, and heavily interested in, a more 
polished version of the KIT. We wanted to use the materials we did have as 
a reference point to which people could react. 

In retrospect, our method of developing and evaluating was closer to a 
problem-solving model in which client needs and developer skills combine to 
produce a product, than to a research-development-dissemination model in 
which the developer incorporates research findings into an instructional 
format whose effectiveness is then tested by user performance. Perhaps we 
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adopted this method because of the tentative nature of the research litera- 
ture on planning and curriculum development; perhapi; the project staff was 
impressed by the complexities of the existing school setting as well as the 
number of decisions required in planning for improved school programs. In 
any case, because of time pressures, our intellectual and professional pre- 
dispositions, and our working st>'ies, we wanted to blend the judgments of 
practitioners with our own. 

One Pilot Test Experience 

The first school in the pilot test was composed of more than 90% rvinor- 
ity students and according to the principal, "All the problems of society 
are in this school." We had two planning teams in this school. 

We provided the teams with materials for the teaching models strategy, 
which consisted of a large blue loose-leaf binder, containing approximately 
100 typed pages, and a set of six descriptions of teaching models printed 
on half sheets and averaging about 20 pages apiece. The loose-leaf binder 
was divided into sections labeled Introduction, the Program Plan, Descrip- 
tion of the Program Plan, and Agendas for planning meetings. 

The first day of this pilot test seemed to result in catastrophe. After 
seeing the materials, one of our two planning teams decided to drop out of 
the piloi test— they even tried to influence the second team to do likewise. 
The team that decided to drop out was composed of experienced teachers. How- 
ever, the idea of planning programs using a "big blue book" that was "complex" 
and "hard to understand" overwhelmed them. They were completely unprepared 
to follow the directions for leading themselves through the planning process. 
None of the team members wanted to prepare for, or lead, a planning meetfng. 
They were angry when they found out that in order to follow our planning 
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procedures they would have to "fill out more forms." Although we thought we 
had explained ourselves clearly, the teachers felt we had misrepresented sys- 
tematic planning both in terms of the amount of work involved and in what 
they would achieve through their efforts. They said: "We need a system to 
enforce discipline. . . to communicate with administrators, . . to get the 
materials and resources we need to teach." They wanted a system to solve 
their problems. What we seemed to offering was more paper. 

In the process of discussing whether to drop out, the team spent a lot 
of time telling the CSE observer about the difficulties and strains of teaching 
in a school with problems. 

There is no discipline here. It's not that the kids are vicious or 
violent. They are really good kids. It's that there is so much to 
overcome, and all my energy is spent just trying to keep the lid on 
things. . . We don't have any facilities here that do us any good. 
We can't get into the library. We don't have any teacher aids, and 
what money we do have for materials is being spent in the lower 
grades. This in-service thing isn't what we were told it was. It 
can't be done in the time we are supposed to do it in . . . There is 
no solution to the problems we have at this school. Or at least the 
solutions are not in your blue book. I probably would have been 
interested and excited about going through these materials when I 
taught at School. These materials won't do me any good here. 

At the end of this first meeting, the team turned in their notebooks. 

We reacted to this turn of events with desperation. We wanted someone 
out there in the real world to react to our materials. We had spent three 
months getting into our pilot test schools and making arrangements for in- 
service credit, and after the first day of pilot testing, we had lost one 
of our two teams. There was also a good chance that the second team would 
follow the first. We contacted the principal and the members of both plan- 
ning teams. We promised that there would be no additional work for the 
teachers beyond the meetings for which they were receiving credit, end that 
we would tailor the planning meetings to their needs • We stressed the 



ERIC 



8 



sincerity of our desire to have their ideas for the development of our ma- 
terials. At this point, our plans for controlling the conditions of the 
pilot test and the collection of data were discarded. 

We did convince the members of the team to return and the members of the 
other team to remain in the pilot test. Our subsequent experience with the 
two teams was similar, but we will focus on the "drop out" team so as to pro- 
vide a specific illustration of how this team, perhaps more than any other, 
demonstrated to us the importance of listening to product users and under- 
standing their point of view. At this early stage of pilot testing, we came 
to realize that we needed communication skills related to receiving as well 
as transmitting information. 

The second meeting was smoother than the first. The teachers spent much 
of the meeting communicating to th':^ CSE observer what it was like to teach 
in a problem school. One teacher told of an experience that had just occurred 
that day. 

I have a boy in my class who was kicked out of school today. I am 
very upset about what happened because the kid is on the verge of 
going to pieces. His father, whom he adored, was murdered three 
weeks ago, and his mother is just getting out of the hospital. I 
have been doing everything I could think of to help him through his 
hurt, and have tried hard to make exceptions for him. Then along 
comes this other teacher who doesn't know what the situation is. 
Apparently the boy badmouthed this other teacher and she suspended 
him and told him, 'Just wait until I tell your father what you said 
to me.' I have argued for a long time that kids shouldn't be sus- 
pended without consulting the pupil's teacher. This kind of thing 
just shouldn't happen, and all I could do is just tell the kid that 
I was really sorry, I felt like crying for him. 

As the discussion gradually focused on the task at hand, some time was 
spent identifying the goal for which the planning team wanted to plan a pro- 
gram. Although the principal had told us that a needs assessment had already 
been done, the planning team teachers were either unaware of it or they dis- 
agreed with the results. In any case it was obvious to both the team and our 



9 



staff observer that before planning could profitably begin, there had to.be 
some kind of agreement on o program goal. 

After being assured that changes in the materials would be made as a 
result of their comments from the first meeting, the team finally began to 
discuss the KIT itself. The team was told that entire sections would be 
eliminated (the introductory materials had particularly caused the team prob- 
lems), and the remaining sections would be shortened and put into a different 
format. They seemed pleased with the stated changes. They tried to soften 
their criticisms and expressed their desire to provide the observer with 
suggestions "that would make the materials better so they wouldn't turn off" 
future users. 

The staff observer and the team then jointly decided that instead of 
working independently, the planning process would be led by the CSE obser- 
ver. For the first time in the pilot test, the team began to read the ma- 
terials purposefully. A two-way relationship began to grow between the team 
and the observer. Something of the feeling of this growing relationship is 
reflected in the concluding remarks of the observer in his pilot test report 
for the second planning meeting. 

This group approaches tasks with great intensity and much fueling. They 
want us to know about the 'real' world of teachers. They spend a great 
ceal of time relating their experiences in teaching. They are a marve- 
lous source of anecdotes and insights into what it is like to try and 
survive as a teacher in a difficult school. They want to help and im- 
prove our materials, but they tend to move slowly and nitpick at times. 
I believe this is because they do not completely understand or believe 
materials like ours can be helpful. They are good critics from the 
point of view of teachers. Their crap detectors never seem to turn off. 
The relationship I have with the team seems to be growing in a positive 
direction, I believe we respect and like one another. 

By the third planning meeting, the project staff had begun rewriting 

and repackaging the materials. We were using the suggestions of each of the 

planning teams in the school, going back to our typewriters, and developing 
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cr rewriting mater'^als based on the recommendations of our pilot test partici- 
pants. The teacher*^ approved of the changes being made in the materials and 
seemed surprised at the seriousness with which we took their opinions. 

During the third planning meeting, 1t became obvious to the observer 
that the materials did not, ev^n when revised, meet the expressed needs of 
the teachers. They talked of problems • elated tz lack of supplies, inac.- 
quate physical plant, poor communications, lack of teacher aides, and a gene- 
ral lack of funds. Clear''v, their major concerns were not related to the 
curricular and inscructional issues for which our materials were appropriate. 
However, after the observer acknowledged that the materials did not seem to 
fit their needs, team members of rered additional comments and observat ^'ons 
about the materials. They reported that they found the use of teaching models 
in their classrooms to be an interesting idea- They suggested that organi- 
zing the materials in terms of meeting agendas resulted in "too rigid an approach." 
Ihey suggested that the materials be organized by major steps or activities - 
because teachers would have to "steal" whatever time they could and organize 
and schedule meetings to fit their schedules. They then volunteered to help 
the observer by going through their notebooks and removing aii the pages 
they thought were irrelevant"-lead"*^g to removal of about half the pages. 

At this point, the CSE staff member told the teachers that we had been 
considering the development of a problem solving planning strategy that might 
be more useful to them than the teaching model strategy. The team immediately 
agreed that a problem solving strategy sounded more relevant. It was decided 
that the remaining two Meetings in the pilot test would be used to explore 
that strategy. The team members retained the teaching models materials, how- 
ever, and at the fourth planning meeting they began by spontaneously repor- 
ting to the CSE jtaf^ member how they would reorganize the teaching models 
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materials. They suggested simplifying the la":guage and eliminating "theo- 
retical" sections. On their own time they had bean reading the descriptions 
of teaching models and were enthusiastic about them^ suggesting that they 
should be introduced early in the planning process. 

It was evident that the team members were now doing more than was asked 
of them and reading the materials voluntarily so they could make their crit- 
icisms. In his pilot test report the observer wrote: 

What I find interesting about the team's comments is the apparent 
change of heart from the first meeting. At the first meeting they 
said they really didn't want to read anything (or very little). 
They particularly said that they did not want to read at horne. Let 
me hypothesize about this apparent change. First we have had an 
opportunity to develop a relationship. I am not just another out- 
sider to them anymore. I think they believe that I care about them 
and their situation. They in turn want to be helpful to me and have 
an investment in the materials in that they have really been trying 
to help and improve them. . . 

The remainder of the fourth planning meeting wa5; devoted to formulating 
a problem solving strategy. Although the problem solving strategy was ultim- 
ately eliminated from the KIT, the efforts of this planning team in working 
with the strategy were relevant and helpful to the later development of the 
Coordinator's Handbook. 

In the fifth and final meeting the team again reported back to the ob- 
server on their reactions to teaching model materials read between meetings. 
By this time they had voluntarily read all the descriptions of the teaching 
models. They said, "We really enjoyed reading them." They were able to 
discuss them intelligently demonstrating that they were aware of many of the 
difficulties involved in implementing teaching models in schools. They 
pointed out problems that our staff had not anticipated. 

In retrospect it was our good fortune that we were able to talk this 
planning team into continuing in the pilot test. We emerged from the ex- 
perience convinced that we had information to make the KIT more useful. Yet, 
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in the five sessions we held with this team they seldom used the materials 

as v?e originally intended them to be used, nor did they fill out our forms. 

What they did do was discuss their frustrations, problems, and points of view. 

They, not we, defined the rules of the gamp. As the observer stated in his 

last report for this team: 

These teachers seem to have needed someone to talk to who would care 
and listen. In reflecting back on the experience I am positive that 
they would have thrown me out of the school had I not listened to 
their problems. I was a good listener in large part because they 
forced me tc listen by threatening to drop out if I did not. 

In general this planning team forced us ta face the fact that teachers 

working in schools have problems and needs that we outside the schools do not 

fully attend to. We decided later that some of the problems v:ere beyond our 

control, but others we tried to deal with once they were brought to our atten- 

tion. Perhaps most important, these teachers taught us that there are people 

"out there" who have relevant important things to say and that if we hope to 

produce products to meet their needs, we should listen to what they have to 

say. 

What we learned about the KIT 

In this section we will list some of the conclusions we reached concer- 
ning the KIT and Its development based on this and .subsequent pilot tests. 
Following this we will describe some of the communication difficulties we 
encountered, and what we did about them. In the last section we will mention 
other writers who share our concern. 

From our experiences with this team and others which participated in 
later pilot tests, we received confirmation for some of our original assump- 
tions. 

1. A needs assessment must precede program planning. The planning 
team must be in agreement on program goals before it is possible 
for them to plan a program. 



13 



2. PV::nning programs at the local school level is an idea v;hich is 
enthusiastically endorsed by teachers. 

3. Planning in teacher teams is welcomed by teachers, but only if 
certain conditions exist. These are the availability of time 
and an assurance that teachers have some power; that is, that 
decisions made by teachers will count. 

4. Teachers do not have systematic planning skills. Motivating 
teachers to plan systematically ..lay be a task that requires 
more than the provision of self-instructional materials. 

5. Alternative planning strategies seem desirable. Clearly the 
planning strategy we originally provided for one of the plan- 
ning teams in this pilot test was inappropriate. 

We received directions for further development decisions. 

1. Someone is needed to prepare for and facilitate the planning 
effort within the school. The principal seems to have the poten- 
tial to be ^ leader and facilitator. Certainly no program plan- 
ning effort can succeed without his cooperation, and so there 
must be a role for the principal in any workable program planning 
system. Materials hud to be developed for the principal or his 
designate. The Coordinator's Handbook was the v^esult. 

2. The physical appearance of materials is important. Bulky materials 
discourage the teachers. They must look simple and easy to read. 
Based on this, we developed a small Procedures Booklet and put all 
supplementary materials in a separate Resource Book. 

3. Leaders on the planning team need easy-to-follow directior 
Leaders must be reassured that they need not be "experts" or do 
excessive homework in preparation for leadership roles. We de- 
veloped Leader Guide Cards that help the leader prepare for and 
and conduct each meeting. 

4. Organizing the planning tasks in a sequence of meetings is too 
restrictive. To be practical and flexible the teachers will 
have to determine the content for each meeting and do the tasks 
at their own rate. Consequently, we organized the Procedures 
Booklet by sets of related tasks. 

In addition to the above conclusions, we also developed some tentative 

ideas about methods of data collection to use throughout our pilot testing 

program. 



What we learned about comunication processes 

Although our original intentions were to "listen" to participants in our 
pilot test, we did not anticipate some of the -^^fficul ties which we would 
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encounter. With hindsight, we see that these difficulties seemed to arise 
from differences between ourselves acting in our role as product developers 
and the pilot test participants acting in their role as teachers. The dif- 
ficulties were compounded both by our initial unawareness of their source 
and then by our lack of previous training in dealing with them once we became 
aware of them. 

As product developers, we were comfortable with our kind of written and 
spoken language. Our materials and our oral presentations were full of ab- 
stractions, generalization and definitions. The teachers, on the other hand, 
were comfortable with more concrete language (one teacher noted wryly the 
"child-speak" which often characterizes the sentence structure and vocabulary 
of elementary school teachers) and with details, anecdotes and references to 
specific children or situations. In order to send our message, we had to 
overcome the language barrier. If we wanted to talk to teachers effectively, 
we had to translate our language into theirs rather than decry their inability 
to understand ours. We found that our skill in doing this, even after we 
realized the problem, was minimal. We realized that we needed to triiin our- 
selves to change language systems. 

Underlying the language difficulties was a deeper difference between the 
teachers and ourselves in perspective and purpose which interfered with our 
communication. Our perspective was long range, theirs short range. We were 
presenting a long term, indirect solution to what teachers regarded as im- 
mediate daily concerns. The fact that they had not yet found solutions to 
their immediate problems did not convince them that they should change their 
strategy, but merely intensified their search for quick answers. We, on the 
other hand, assumed as self-evident the practicality and rationali^ty of long 
range solutions, and did not have arguments in our repetoire which made our 



15 

case convincing to the teachers. We realized that we needed to build a case 
for our view that would make sense within the context of the teachers' exper- 
ience. 

In addition, our purposes for the pilot test were different from those 
of the teachers. Our primary purpose in conducting pilot tests was to have 
teachers help us improve our materials. We indicated this to principals and 
teachers in soliciting their help for the pilot test, but we also stressed 
that^ in exchange for their time, the materials themselves would help them 
acquire knowledge and skills of use to them. Since we had arranged for in- 
service credit, the teachers expected that they would learn techniques which 
would be of immediate use to them in the classroom as they did in other in- 
service courses. They were not prepared to work very hard on their own. 
Although they knew about our revision needs, they were not especially inter- 
ested in spending their time accommodating them. They wanted help with their 
own work problems. Their initial skepticism that researchers from a univer- 
sity setting could provide anything of practical use to them was confirmed 
when they saw the voluminous materials we had prepared. Correspondingly, the 
more resistant they were, the greater their difficulty in focusing on any part 
of our materials which might be useful to them. We realized that we needed 
to rethink the pilot test experience so it would be a worthwhile one for them 
as well as for us. 

In an effort to open channels of cominunication and meet some of these 
difficulties, we tried the following: 
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1. Listened empathetical ly and tried to understand the situation from 
the teachers* point of view.* 

2. Encouraged the tf?achers on the planning team to talk about their 
problems even when such problems appeared to be irrelevant to the materials. 
{Me often found these discussions gave us clues about contextual factors-- 
particularly social ones--that might influence the use of the KIT.) 

3. Encouraged the teachers on the planning team to use our materials 
or talk to us about then? on their own terms-^even if this iT^eant that they 
never tried to use or discuss the materials in the way that we originally 
intended. (This non-directive approach often brought out and emphasized 
problems in the materials, or allowed the teachers to show us more workable 
procedures to build into the materials.) 

4. Rewarded participants for their help. Sometimes we volunteered what 
amounted to consultant services in our efforts to help teachers with some of 
their problems. (Although what we offered the participants was limited, we 
believe that our gestures were important.) 

5. Spent as much time as possible with the participants. The same staff 
member was assigned to a particular planning team for the duration of the pi- 
lot test. (This provided the observer with the opportunity to become well 
acquainted with the personalities, problems, and point of view of each plan- 
ning team. ) 

♦Rogers (1969) described empathetic understanding as follows: 

This kind of understanding is sharply different from the usual evaluative 
understanding, which follows the pattern of 'I understand what is wrong 
with you.' When there is a sensitive empathy, however, the reaction . . . 
follows somethinq of this pattern, 'At last someone understands how it 
feels and seems to be me without wanting to analyze me or judge me. 
Now I can blossom and grow and learn (pp. 111-112). 
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6. Became professionally and even socially close to tht* participants. 
During the course of meeting with a group of pilo*^ tost participants we often 
formed temporary relationships akin to friendships. Information was shared 
about families, jobs, goals, hopes, and frustrations. (We found that the 
development of close, personal relationships generally permitted the partici- 
pants to communicate their perceptions to us more openly and effectively.) 

7. Took extensive descriptive notes of what occurred during each meet- 
ing attempting to capture the participantf* point of view. We tried to write 
low inference descriptions of each meeting, recording participants' words and 
actions as accurately as possible. (We found our records quite helpful for 
our formative evaluation purposes. It was not at all difficult to go back 
over our notes and pick out the problems that seemed to arise frequently.) 

Our experience with the pilot test of the Program Planning KIT has led 
us to two propositions: 

1. With products designed to bj used by teachers in school settings, 
informal^ relatively unstructured, pilot testing is a productive 
method for gathering information to guide development decisions. 

2. The effectiveness of such informal pilot tests depends on the de* 
gree of communication established between evaluator/developer and 
product user. There are barriers to such communication arising 
from real differences in the perspectives of evaluator/developers 
and practitioners. The product developer must acknowledge their 
existence, and must develop skill in overcoming them. For ex- 
ample^ product developers should be sensitive to the differences 
in language which they speak, and the language spoken by teachers, 
and be able to modify their own written and oral language. Product 
developers should be sensitive to the context within which teachers 
operate, and find ways to make the pilot tests a worthwile enter- 
prise for both parties. Product developers need a broad array of 
methodological alternatives for gathering information--a repetoire 
which includes clinical skills of observation, listening, reflecting 
back, and field study skills for recording and interpreting infor- 
mation. 

These propositions are related to current concerns in three areas: those 
of product development and formative evaluation; field research methodology; 
and knowledge utilization and dissemination. The ;ollowing discussion not 
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intended to review the literature, but rather to suggest how our perspective 
relates to what others have written. 

Related areas of concern 

Product development and formative evaluation 

In their recent summary of types of formative evaluation, Sanders and 
Cunningham (1973) characterize those activities that occur early in the pro- 
duct development effort as "formative interim evaluation" activities because 
they deal with pieces of the product rather than the entire assembled pack- 
age. They distinguish between formal and informal formative interim evalua- 
tion and note that in informal eval uation--which is generally unstructured-- 
observations and discussions often uncover critical dimensions of the product 
that might otherwise go unnoticed, 

Mdrkel (1967) described the initial or laboratory stage of product de- 
velopment and noted that the types of data collected as well as the method 
for collection may be more varied during this early stage than at other later 
stages. She suggested that data at tin's stage tend to be clinical in the 
sense that they are the result of close observation and interaction wii;h in- 
dividuals or small groups. She additionaly observed that this clinical stage 
is cften omitted in many product development efforts. 

Baker and Alkin (1973) discussed the types of data that should be col- 
lected during the initial development stage and indicate the importance of 
obtaining information that reveals the effects of the product as well as in- 
formation that leads to a diagnosis of what is wrong with the product. Al- 
though they implied a preference for formal data collection procedures, their 
emphases on ascertaining user attitudes and satisfaction with the product as 
well as their concern with unanticipated side effects sr.lght suggest that 
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more informal procedures are also acceptable. They noted that since the de- 
veloper should expect early drafts of the product to meet with only limited 
success, small numbers of subjects should be involved and they should be ex- 
pected to provide "rich data." 

There seems to be agreement, then, that a variety of means for gathering 
information at the early stages of development of a product is desirable. 
The decision as to whether to use formal or informal procedures (or both) may 
depend upon the use to which the developer wishes to put the information. 
If the information is to bt: used to revise a product whose specifications are 
fixed--that is, if only the means of instruction rather than the instructional 
goals are to be changed--then more formal pretest and posttest instruments may 
be appropriate. If, on the other hand, the information is to be used to make 
decisions about the goals themselves (and to learn as much as possible about 
the context within which the goals are being introduced and received) then 
more informal methods are appropriate. 

Field Research Methodology 

Clinical procedures for gathering information, and methods for recording 
non-standardized information seem far removed from an experimental research 
tradition that relies on control of variables, presentation of stimuli, and 
quantifiable responses. It is clear that this experimental tradition is not 
the only one that has a contribution to make to formative evaluation. There 
are other ways of obtaining information and of making inferences from it. 
Listening to and observing people in real life settings, and recording their 
comnents and interractions makes use of techniques coming not from the exper- 
imental or measurement tradition but from the field research tradition of 
sociology and anthropology. For instance, Schatzman and Strauss (1973) 
stated: 
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For the naturalistically oriented humanist, the choice of method 
is virtually a logical imperative. The researcher must get close 
to the people whom he studies; he understands that their actions 
are best comprehended when observed on the spot--in the natural, 
on-going environment where they live and work. If man creates at 
least some of the conditions for his own actions, then it can be 
presumed that he acts on his own world, at the very place and 
time that he is. The researcher himself must be at the location, 
not only to watch but also to listen to the symbolic sounds that 
characterize this world. A dialogue with persons in their natural 
situation will reveal trie nuances of meanings from which their 
perspectives and definitions are continually forged (pp. 5-6). 

Recording such observations obviously cannot be done on a structured 
form. Lofland (1971) noted that field notes or reports should be written by 
a researcher who has had physical and social proximity for a significant 
period of time with the people on whom he is reporting. He notes that such 
reports should be characterized by attention to detail; truthfulness irv that 
the reporter should accurately describe what he sees and hears; and direct 
quotations from participants to capture the reality of the situation as they 
perceive It. In field notes such as these, inferences should be kept sepa- 
rate, in so far as possible, from description. 

Knowledge Utilization and Dissemination 

Many writers have recently expressed concern about the gap that continues 
to exist between practitioner and researcher. Baldridge (1973) has indicated 
that the real differences in perspective and orientation between these two 
groups have been incorporated into myths and stereotypes thus widening the 
gulf between the two groups and making open communication even more difficult. 
He notes: 

In the mythology of the field user, the researcher is an unfeeling 
egghead with computer printout in hand who advances impractical 
theories to schools populated with random samples. From the oppo- 
site viewpoint, the myth portrays a non-intellectual short-sighted 
person, bogged down in a world of lesson plans, report cards and 
Dick and Jane. These extreme, negative stereotypes disrupt or 
prevent relationships that might otherwise be beneficial (pp. 15-16) 
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A recent policy paper by the National Institute of Education (1973) notes 
that the "customary" product development rnodel projects a "passive, compliant 
role for the school based consumer," and that this approach assuu^es that the 
schools will faithfully adopt and install research and development products 
in some uniform manner provided only that the research and developnnent commu- 
nity demonstrates that they work. Among the difficulties with this view, the 
paper suggests, is that: 

It sees educational practice--or at least innovative educational 
practice--as a simple function of external research, development 
and linkage. Yet, what actually goes on in schools is a function 
of many others influences as well, including traditional lore, and 
craft knowledge, the perceptions practitioners have of the success 
or failure of trial and error innovation, and the particular needs 
and circumstances of special schools and communities (p. 53). 

Early user involvement in the development of products is no guarantee 
that products responsive to the needs of the practitioner will emerge. Simi- 
larly, open communication between developer and practitioner in early pilot 
tests of materials is no guarantee that developers and users will come closer 
to seeing one another's point of view. Although it is not a sufficient condi- 
tion for bringing developers and practitioners together in a mutually satis- 
factory relationship, recognition by developers of the desirability of accept- 
ing user opinions and judgments in product development enterprises (as well as 
the training of developers in the procedures for establishing such relation- 
ships) seems to be a necessary step. 
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